Repository navigation
feat: worker generic error retry + DB test coverage + error detail logging - #1439
Conversation
…coverage Worker: transactions that fail with generic exceptions now get exponential backoff (10s base) and cancel after 3 retries instead of silently retrying forever. Health endpoint exposes retry count for observability. Tests: 25 new DB integration tests covering update_transaction_status, consensus_history, set_transaction_result, timestamps, appeals, state_timestamp, get_transactions, contract_snapshot, and rotation_count.
Add `detail` field to GenVMInternalError carrying the raw error string (truncated to 1000 chars). Worker logs now include the full Rust ModuleError with ctx (HTTP status, response body from OpenRouter), so we can see *why* LLM calls fail instead of just "LLM_EXECUTION_ERROR".
📝 WalkthroughWalkthroughAdded a generic-error retry mechanism with exponential backoff and max-retry limits to the consensus worker; GenVM INTERNAL_ERROR now carries an optional detail string; worker health now reports active generic-error retries; JSON-RPC GenVM calls are concurrency-limited; tests and docker-compose adjustments included. Changes
Sequence DiagramsequenceDiagram
participant Client as Client
participant TxWorker as ConsensusWorker
participant GenVM as GenVM Executor
participant State as State/DB
participant WS as WebSocket Notifier
Client->>TxWorker: Submit transaction
TxWorker->>GenVM: Execute transaction
GenVM-->>TxWorker: GenVMInternalError (with detail)
TxWorker->>TxWorker: _handle_generic_error_retry(tx_hash, error)
alt retries < MAX_GENERIC_ERROR_RETRIES
TxWorker->>TxWorker: increment retry count & set base backoff
TxWorker->>TxWorker: schedule exponential backoff
TxWorker-->>Client: defer / retry later
else retries >= MAX_GENERIC_ERROR_RETRIES
TxWorker->>State: persist tx status = CANCELED with error detail
TxWorker->>WS: notify cancellation
TxWorker->>TxWorker: remove retry tracking
end
Note over TxWorker: On next processing attempt\n_is_in_backoff checks generic-error backoff
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…by_hash compat _prepare_basic_transaction_data expects leader_result[0]["node_config"]["address"] to exist when consensus_results is present. Pass _MockReceipt with proper node_config instead of None.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@tests/db-sqlalchemy/transactions_processor_test.py`:
- Around line 529-535: In test_get_highest_timestamp_ignores_null, the local
variable tx2 is assigned by calling _make_tx(tp) but never used; to silence
static analysis and indicate the value is intentionally unused, replace the tx2
assignment with an underscore (e.g., _ = _make_tx(tp)) or remove the assignment
entirely so the call to _make_tx(tp) is still executed; update the test body
around the _make_tx call in test_get_highest_timestamp_ignores_null accordingly.
🧹 Nitpick comments (5)
backend/node/genvm/error_codes.py (1)
57-62:__repr__does not include the newdetailfield.All other instance attributes (
error_code,causes,is_fatal,is_leader) are surfaced in__repr__, but the newly addeddetailis omitted. This meansrepr(exc)won't show the diagnostic context thatdetailwas specifically added to carry. Consider adding it (perhaps truncated) for debugging consistency.Proposed fix
def __repr__(self) -> str: return ( f"GenVMInternalError(message={self.args[0]!r}, " f"error_code={self.error_code!r}, causes={self.causes!r}, " - f"is_fatal={self.is_fatal!r}, is_leader={self.is_leader!r})" + f"is_fatal={self.is_fatal!r}, is_leader={self.is_leader!r}, " + f"detail={self.detail!r})" )tests/db-sqlalchemy/transactions_processor_test.py (2)
24-43: Missing type hints on_make_txhelper.Per coding guidelines, all Python code should include type hints. The function parameters and return type are untyped.
Proposed fix
-def _make_tx(tp, **overrides): +def _make_tx(tp: TransactionsProcessor, **overrides: object) -> str: """Create a transaction with sensible defaults and unique hash."""As per coding guidelines: "Include type hints in all Python code".
424-430:_MockReceiptlacks type hints andstrip_contract_stateis intentionally unused.The unused parameter mirrors the real
Receipt.to_dictinterface, which is fine. Adding a type hint would satisfy the coding guidelines.Proposed fix
class _MockReceipt: - def to_dict(self, strip_contract_state=False): + def to_dict(self, strip_contract_state: bool = False) -> dict: return {As per coding guidelines: "Include type hints in all Python code".
backend/consensus/worker.py (2)
993-1001: Unboundedstr(error)stored inconsensus_data— consider truncating.
str(error)can be arbitrarily long (e.g., full stack traces or large HTTP response bodies). This is persisted to the DB's JSONB column and returned through APIs. Thedetailfield onGenVMInternalErroralready truncates to 1000 chars; applying similar bounds here would be consistent.Proposed fix
tx.status = TransactionStatus.CANCELED tx.consensus_data = { "error": "max_generic_retries_exceeded", - "last_error": str(error), + "last_error": str(error)[:1000], "retries": retry_info["count"], }
43-43:MAX_GENERIC_ERROR_RETRIESis a class constant while the analogous no-validators limit is env-configurable.This is fine for now, but if operators need to tune retry behavior without redeploying, consider making it env-configurable like
NO_VALIDATORS_MAX_RETRIES.
| def test_get_highest_timestamp_ignores_null(self, tp): | ||
| tx1 = _make_tx(tp) | ||
| tx2 = _make_tx(tp) | ||
| tp.set_transaction_timestamp_awaiting_finalization(tx1, 9999) | ||
| tp.session.commit() | ||
| # tx2 has no timestamp set | ||
| assert tp.get_highest_timestamp() == 9999 |
There was a problem hiding this comment.
Unused variable tx2 — assign to _ or remove the name.
Static analysis flags tx2 as assigned but never used. The intent is to create a second transaction without a timestamp, but the returned hash isn't needed. Use _ to signal this is intentional.
Proposed fix
def test_get_highest_timestamp_ignores_null(self, tp):
tx1 = _make_tx(tp)
- tx2 = _make_tx(tp)
+ _make_tx(tp) # second tx with no timestamp set
tp.set_transaction_timestamp_awaiting_finalization(tx1, 9999)
tp.session.commit()
# tx2 has no timestamp set
assert tp.get_highest_timestamp() == 9999📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def test_get_highest_timestamp_ignores_null(self, tp): | |
| tx1 = _make_tx(tp) | |
| tx2 = _make_tx(tp) | |
| tp.set_transaction_timestamp_awaiting_finalization(tx1, 9999) | |
| tp.session.commit() | |
| # tx2 has no timestamp set | |
| assert tp.get_highest_timestamp() == 9999 | |
| def test_get_highest_timestamp_ignores_null(self, tp): | |
| tx1 = _make_tx(tp) | |
| _make_tx(tp) # second tx with no timestamp set | |
| tp.set_transaction_timestamp_awaiting_finalization(tx1, 9999) | |
| tp.session.commit() | |
| # tx2 has no timestamp set | |
| assert tp.get_highest_timestamp() == 9999 |
🧰 Tools
🪛 Ruff (0.15.0)
[error] 531-531: Local variable tx2 is assigned to but never used
Remove assignment to unused variable tx2
(F841)
🤖 Prompt for AI Agents
In `@tests/db-sqlalchemy/transactions_processor_test.py` around lines 529 - 535,
In test_get_highest_timestamp_ignores_null, the local variable tx2 is assigned
by calling _make_tx(tp) but never used; to silence static analysis and indicate
the value is intentionally unused, replace the tx2 assignment with an underscore
(e.g., _ = _make_tx(tp)) or remove the assignment entirely so the call to
_make_tx(tp) is still executed; update the test body around the _make_tx call in
test_get_highest_timestamp_ignores_null accordingly.
…ilation GenVM precompiles 6+ WASM modules on first start (~18s each = ~108s total). The previous 40s start_period caused healthcheck failures in CI before the Flask app could even start.
Commit 621a568 changed HealthCache.genvm_healthy default from True to False. The readiness integration test has no GenVM manager, so the check short-circuits to 503 before evaluating the RPC router. Set the cache flag explicitly so the test exercises the intended code path.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/db-sqlalchemy/transactions_processor_test.py (1)
1-1:⚠️ Potential issue | 🟡 MinorRename to
test_transactions_processor.pyto follow pytest auto-discovery conventions.The file uses the
*_test.pypattern, but pytest's default discovery only recognizestest_*.pyandTest*.pypatterns. No custom pytest configuration was found in the repository. The coding guidelines requiretest_<feature>.pynaming for pytest auto-discovery. Rename totest_transactions_processor.py.
🧹 Nitpick comments (2)
tests/db-sqlalchemy/transactions_processor_test.py (2)
21-43: Global_tx_countercan leak state across test runs in the same process.If pytest runs multiple test files in the same process (the default),
_tx_counteraccumulates across modules and never resets. This isn't a correctness bug today since hashes remain unique, but it makes test isolation fragile. Consider using a pytest fixture oritertools.count()instance scoped to conftest instead.
463-478:sessionfixture parameter is unused in these test methods.Both
test_update_consensus_history_empty_initialandtest_update_consensus_history_appends_roundsacceptsessionbut only usetp(which already owns a session). Safe to remove the extra parameter to silence static analysis.Proposed fix
- def test_update_consensus_history_empty_initial(self, tp, session): + def test_update_consensus_history_empty_initial(self, tp): tx_hash = _make_tx(tp)- def test_update_consensus_history_appends_rounds(self, tp, session): + def test_update_consensus_history_appends_rounds(self, tp): tx_hash = _make_tx(tp)
gen_call had no concurrency limit, allowing unbounded concurrent GenVM executions that cause uvloop file descriptor registry collisions (RuntimeError: File descriptor N is used by transport). Workers already use asyncio.Semaphore(8) in consensus/base.py; this adds the same limit to the jsonrpc gen_call path. Configurable via GENVM_MAX_CONCURRENT.
Replace worker_id-based genlayer_needed_workers metric with per-contract schedulable work measurement. Old metric counted distinct worker_ids from last hour (supply-based, double-counted during rolling upgrades). New metric counts distinct contracts with occupied (in-flight) or runnable (pending, no in-flight) transactions — directly measuring max useful parallelism since transactions per contract are processed sequentially. New gauges: genlayer_occupied_contracts, genlayer_runnable_contracts
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@backend/protocol_rpc/health.py`:
- Around line 1151-1169: The metrics SQL in the per_contract CTE currently
filters statuses to ('PENDING','PROPOSING','COMMITTING','UNDETERMINED') which
diverges from the consensus health check’s broader processing set; either expand
the status list in the CTE to match the consensus health-counting states
(include 'ACTIVATED','REVEALING','ACCEPTED') so occupied/runnable calculations
(SELECT ... AS occupied, ... AS runnable) align with the health/debug endpoints,
or add a clear comment above the query explaining why the narrower set is
intentionally used for HPA scaling decisions; also replace the stray
logging.exception() call with the module logger.exception() used elsewhere to
keep logging consistent.
🧹 Nitpick comments (2)
backend/protocol_rpc/health.py (2)
1154-1161:to_addresscan be NULL — deploy transactions will be grouped into a single bucket.Other queries in this file (e.g., Line 526) filter
WHERE to_address IS NOT NULL. Without that filter, all deploy transactions (NULLto_address) collapse into one group, which is mostly harmless but may slightly skew the occupied/runnable counts.Consider adding
AND to_address IS NOT NULLto the CTE's WHERE clause for consistency.Proposed fix
FROM transactions - WHERE status IN ('PENDING', 'PROPOSING', 'COMMITTING', 'UNDETERMINED') + WHERE status IN ('PENDING', 'PROPOSING', 'COMMITTING', 'UNDETERMINED') + AND to_address IS NOT NULL GROUP BY to_address
1204-1205: Uselogger.exceptioninstead oflogging.exception.Line 1205 uses the module-level
logging.exception(...)while the rest of the file consistently uses theloggerinstance defined on Line 18. This bypasses the module's logger name, making log filtering/routing harder.Proposed fix
- logging.exception("Metrics endpoint failed") + logger.exception("Metrics endpoint failed")
| row = conn.execute( | ||
| text( | ||
| """ | ||
| WITH per_contract AS ( | ||
| SELECT | ||
| to_address, | ||
| BOOL_OR(status = 'PENDING') AS has_pending, | ||
| BOOL_OR(status IN ('PROPOSING', 'COMMITTING', 'UNDETERMINED')) AS has_inflight | ||
| FROM transactions | ||
| WHERE status IN ('PENDING', 'PROPOSING', 'COMMITTING', 'UNDETERMINED') | ||
| GROUP BY to_address | ||
| ) | ||
| SELECT | ||
| COALESCE(COUNT(*) FILTER (WHERE has_inflight), 0) AS occupied, | ||
| COALESCE(COUNT(*) FILTER (WHERE has_pending AND NOT has_inflight), 0) AS runnable | ||
| FROM per_contract | ||
| """ | ||
| ) | ||
| ) | ||
| ).fetchone() |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# First, let's check the file size and read the relevant sections
wc -l backend/protocol_rpc/health.pyRepository: genlayerlabs/genlayer-studio
Length of output: 107
🏁 Script executed:
#!/bin/bash
# Check the status usage around line 523 (the consensus health check)
sed -n '515,535p' backend/protocol_rpc/health.pyRepository: genlayerlabs/genlayer-studio
Length of output: 972
🏁 Script executed:
#!/bin/bash
# Check the status usage in the metrics query (lines 1144-1176)
sed -n '1144,1176p' backend/protocol_rpc/health.pyRepository: genlayerlabs/genlayer-studio
Length of output: 1702
🏁 Script executed:
#!/bin/bash
# Check the logging inconsistency around line 1205
sed -n '1200,1210p' backend/protocol_rpc/health.pyRepository: genlayerlabs/genlayer-studio
Length of output: 534
🏁 Script executed:
#!/bin/bash
# Search for all transaction status checks in the file to understand the full context
rg -n "status\s*(?:IN|=|!=)" backend/protocol_rpc/health.py -A2 -B2Repository: genlayerlabs/genlayer-studio
Length of output: 8095
Status set in metrics query differs from health checks; clarify intent.
The metrics query (lines 1160, 1158) uses ('PENDING', 'PROPOSING', 'COMMITTING', 'UNDETERMINED') to measure occupied and runnable contracts, but the consensus health check (line 523) and debug endpoint (line 1024) count ('ACTIVATED', 'PROPOSING', 'COMMITTING', 'REVEALING', 'ACCEPTED', 'UNDETERMINED') as processing transactions. If transactions spend non-trivial time in ACTIVATED, REVEALING, or ACCEPTED states, the metrics endpoint will undercount occupied contracts and could cause HPA to scale down prematurely.
Consider aligning the status set with the broader definition used in consensus health checks, or add a comment explaining why the narrower set is intentional for HPA scaling decisions.
Additionally, line 1205 uses logging.exception() instead of the logger.exception() pattern used elsewhere in the file (e.g., line 887).
🤖 Prompt for AI Agents
In `@backend/protocol_rpc/health.py` around lines 1151 - 1169, The metrics SQL in
the per_contract CTE currently filters statuses to
('PENDING','PROPOSING','COMMITTING','UNDETERMINED') which diverges from the
consensus health check’s broader processing set; either expand the status list
in the CTE to match the consensus health-counting states (include
'ACTIVATED','REVEALING','ACCEPTED') so occupied/runnable calculations (SELECT
... AS occupied, ... AS runnable) align with the health/debug endpoints, or add
a clear comment above the query explaining why the narrower set is intentionally
used for HPA scaling decisions; also replace the stray logging.exception() call
with the module logger.exception() used elsewhere to keep logging consistent.
|
🎉 This PR is included in version 0.99.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Summary
_transaction_contextnow get exponential backoff (10s base) and cancel after 3 retries instead of silently retrying forever. Health endpoint exposesgeneric_error_retriescount.GenVMInternalErrornow carries adetailfield with the raw error string (truncated 1000 chars) so worker logs show the actual upstream HTTP status/response from OpenRouter instead of justLLM_EXECUTION_ERROR.Test plan
docker compose -f tests/db-sqlalchemy/docker-compose.yml --project-directory . run --build --rm tests)generic_error_retriesfielddetail=ModuleError { causes: [...], ctx: { status: 429, ... } }on GenVM errorsSummary by CodeRabbit
New Features
Bug Fixes
Chores
Tests